The present invention relates to audio-video transmitter and audio-video receiver, data-processing apparatus and method, waveform-data-transmitting method and apparatus and waveform-data-receiving method and apparatus, and video-transmitting method and apparatus and video-receiving method and apparatus.
There has been an apparatus which satisfies the sense of real existence that a counterpart is present in front of you and aims at realistic picture communication by extracting, for example, a person""s picture out of the scenery picture of a space in which you are present and superimposing the person""s picture, a person""s picture sent from the counterpart, and the picture of a virtual space to be displayed commonly with a previously-stored counterpart on each other and displaying them (Japanese Patent Publication No. 4-24914).
Particularly, in the case of the prior art, inventions concerned with acceleration for performing picture synthesis and a method for reducing memories are made (e.g. Official gazette of Japanese Patent Publication No. 5-46592: Picture synthesizer).
Though a communication system using picture synthesis for synthesizing two-dimensional static pictures or three-dimensional CG data has been proposed by the prior art, specific discussion on a method for realizing a system for simultaneously synthesizing a plurality of video(picture) and a plurality of audio and displaying them has not been performed from the following viewpoints.
That is, there has been a problem that no specific discussion has been performed from the following viewpoints:
(A1) a method for transmitting (communicating and broadcasting) and controlling pictures and audio under the environment in which data and control information (information transmitted by a packet different from that of data to control the processing of terminal side) are independently transmitted by using a plurality of logical transmission lines constructed by software on one real transmission line or more;
(A2) a method for dynamically changing header information (corresponding to data control information of the present invention) to be added to data for a picture or audio to be transmitted;
(A3) a method for dynamically changing header information (corresponding to transmission control information of the present invention) to be added for transmission;
(A4) a method for transmitting information by dynamically multiplexing and separating a plurality of logical transmission lines;
(A5) a method for transmitting pictures and audio considering the read and rise periods of program or data; and
(A6) a method for transmitting pictures and audio considering zapping.
However, the method for changing encoding systems and a method of discussing data in frames in accordance with the frame type of a picture have been proposed so far as a method for dynamically adjusting the amount of data to be transmitted to a network (H. Jinzenji and T. Tajiri, A study of distributive-adaptive-type VOD system, D-81, System Society of Institute of Electronics, Information and Communication Engineers (IEICE) (1995)).
A dynamic throughput scalable algorithm capable of providing a high-quality video under a restricted processing time is proposed as a method for adjusting throughput at the encoder side (T. Osako, Y. Yajima, H. Kodera, H. Watanabe, K. Shimamura: Encoding of software video using a dynamic throughput scalable algorithm, Thesis Journal of IEICE, D-2, Vol. 80-D-2, No. 2, pp. 444-458 (1997)).
Moreover, there is an MPEG1/MPEG2 system as an example of realizing synchronous reproduction of video and audio.
(B1) The conventional method for discussing a picture correspondingly to the frame type of the video has a problem that it is difficult to preponderantly reproduce an important scene cut synchronously with audio by handling a plurality of video streams or a plurality of audio streams and reflecting the intention of an editor because the grading of the information which can be handled is in a single stream.
(B2) Moreover, it must be possible that a decoder decodes every supplied bit stream because it is a prerequisite that MPEG1/MPEG2 is realized by hardware. Therefore, it is a problem how to correspond to the case of exceeding the throughput of the decoder.
Moreover, to transmit video, there have been some systems including a system such as H. 261 (ITU-T Recommendation H. 261-Video codec for audio-visual services at px 64) and they have been mounted by hardware. Therefore, the case has not occurred that decoding is not completed within a designated time because of considering the upper limit of a necessary performance when designing hardware.
The above-designated time denotes a time required to transmit a bit stream obtained by coding a sheet of video. If decoding is not completed within the time, an extra time becomes a delay. If the delay is accumulated, the delay from the transmitting side to the receiving side increases and the system cannot be used as a video telephone. This state must be avoided.
Moreover, when decoding cannot be completed within a designated time because a communication counterpart generates an out-of-spec bit stream, a problem occurs that a video cannot be transmitted.
The above problem occurs not only for a video but also for audio data.
However, in recent years, because the network environment formed by personal computers (PCs) has been arranged as the result of spread of internet and ISDN, the transmission rate has been improved and it has been possible to transmit a video by using PCs and a network. Moreover, requests for transmission of video by users have been rapidly increased. Furthermore, a video can be completely decoded by software because CPU performances have been improved.
However, because the same software can be executed by personal computers different in structure such as a CPU, bus width, or accelerator, it is difficult to previously consider the upper limit of a necessary performance and therefore, a problem occurs that a picture cannot be decoded within a designated time.
Moreover, when coded data for a video having a length exceeding the throughput of a receiver is transmitted, coding cannot be completed within a designated time.
Problem (C1)
Decreasing a delay by decoding a picture within a designated time.
When inputting a video as the waveform data of claim C1 of the present invention or outputting a video as the waveform data of claim C7 of the present invention as means for solving the problem 1, a problem may be left that the substantial working efficiency of a transmission line is lowered because a part of a transmitted bit stream is not used. Moreover, there are some coding systems that generate a present decoded video in accordance with a last decoded picture (e.g. P picture). However, because the last decoded picture is not completely restored by the means for solving the problem 1, there is a problem that deterioration of the picture quality influentially increases as time passes.
Problem (C2)
In the case of the means for solving the problem 1, the substantial working efficiency of a transmission line is lowered. Moreover, picture-quality deterioration is spread.
Furthermore, in the case of mounting by software, the frame rate of a picture is determined by the time required for one-time coding. Therefore, when the frame rate designated by a user exceeds the throughput of a computer, it is impossible to correspond to the designation.
Problem (C3)
When the frame rate designated by a user exceeds the throughput of a computer, it is impossible to correspond to the designation.
When considering the problems (A1) to (A6), it is an object of the present invention to provide an audio-video transmitter and audio-video receiver and data-processing apparatus and method in order to solve at least any one of the problems.
Moreover, when considering the problems (B1) and (B2), it is another object of the present invention to provide data-processing apparatus and method in order to solve at least one of the problems.
Furthermore, when considering the problems (C1) to (C3), it is still another object of the present invention to provide waveform-data-receiving method and apparatus and waveform-data-transmitting method and apparatus, and video-transmitting method and apparatus and video-receiving method and apparatus in order to solve at least one of the problems.
The present invention according to claim 1 is an audio-video transmitting apparatus comprising transmitting means for transmitting the content concerned with a transmitting method and/or the structure of data to be transmitted or an identifier showing the content as transmission format information through a transmission line same as that of the data to be transmitted or a transmission line different from the data transmission line; wherein said data to be transmitted is video data and/or audio data.
One aspect of the present invention is the audio-video transmitting apparatus, wherein said transmission format information is included in at least one of data control information added to said data to control said data, transmission control information added to said data to transmit said data, and information for controlling the processing of the terminal side.
Another aspect of the present invention is the audio-video transmitting apparatus, wherein at least one of said data control information, transmission control information, and information for controlling the processing of said terminal side is dynamically changed.
Still another aspect of the present invention is the audio-video transmitting apparatus wherein said data is divided into a plurality of packets, and said data control information or said transmission control information is added not only to the head packet of said divided packets but also to a middle packet of them.
Yet another aspect of the present invention is the audio-video transmitting apparatus, wherein an identifier showing whether to use timing information concerned with said data as information showing the reproducing time of said data is included in said transmission format information.
Still yet another aspect of the present invention is the audio-video transmitting apparatus, wherein said transmission format information is the structural information of said data and a signal which is output from a receiving apparatus receiving the transmitted structural information of said data and which can be received is confirmed and thereafter, said transmitting means transmits corresponding data to said receiving apparatus.
A further aspect of the present invention is the audio-video transmitting apparatus, wherein said transmission format information include (1) an identifier for identifying a program or data to be used by a receiving apparatus later and (2) at least one of a flag, counter, and timer as information for knowing the point of time in which said program or data is used or the term of validity for using said program or data.
Still a further aspect of the present invention is the audio-video transmitting apparatus, wherein said point of time in which said program or data is used is transmitted as transmission control information by using a transmission serial number for identifying a transmission sequence or as information to be transmitted by a packet different from that of data to control terminal-side processing.
Still yet a further aspect of the present invention is the audio-video transmitting apparatus wherein storing means for storing a plurality of contents concerned with said transmitting method and/or said structure of data to be transmitted and a plurality of its identifiers are included, and said identifier is included in at least one of said data control information, transmission control information, and information for controlling terminal-side processing as said transmission format information.
Another aspect of the present invention is the audio-video transmitting apparatus wherein storing means for storing a plurality of contents concerned with said transmitting method and/or said structure of data to be transmitted are included, and said contents are included in at least one of said data control information, transmission control information, and information for controlling terminal-side processing as said transmission format information.
Still another aspect of the present invention is the audio-video transmitting apparatus wherein a default identifier showing whether to change the contents concerned with said transmitting method and/or structure of data to be transmitted is added.
Still yet another aspect of the present invention is the audio-video transmitting apparatus, wherein said identifier or said default identifier is added to a predetermined fixed-length region of information to be transmitted or said predetermined position.
A further aspect of the present invention is an audio-video receiving apparatus comprising: receiving means for receiving said transmission format information transmitted from the audio-video transmitting apparatus; and transmitted-information interpreting means for interpreting said received transmission-format information.
A still further aspect of the present invention is the audio-video receiving apparatus, wherein storing means for storing a plurality of contents concerned with said transmitting method and/or said structure of data to be transmitted and a plurality of its identifiers are included, and the contents stored in said storing means are used to interpret said transmission format information.
A still yet further aspect of the present invention is an audio-video transmitting apparatus comprising: information multiplexing means for controlling start and end of multiplexing the information for a plurality of logical transmission lines for transmitting data and/or control information is included; wherein, not only said data and/or control information multiplexed by said information multiplexing means but also control contents concerned with start and end of said multiplexing by said information multiplexing means are transmitted as multiplexing control information, and said data includes video data and/or audio data.
Another aspect of the present invention is the audio-video transmitting apparatus wherein it is possible to select whether to transmit said multiplexing control information by arranging said information without multiplexing it before said data and/or control information or transmit said multiplexing control information through a transmission line different from the transmission line for transmitting said data and/or control information.
Still another of the present invention is an audio-video receiving apparatus comprising: receiving means for receiving said multiplexing control information transmitted from the audio-video transmitting and said multiplexed data and/or control information; and separating means for separating said multiplexed data and/or control information in accordance with said multiplexing control information.
Yet another aspect of the present invention is an audio-video receiving apparatus comprising: main looking-listening means for looking at and listening to a broadcast program; and auxiliary looking-listening means for cyclically detecting the state of a broadcast program other than the broadcast program looked and listened through said main looking-listening means; wherein said detection is performed so that a program and/or data necessary when said broadcast program looked and listened through said main looking-listening means is switched to other broadcast program can be smoothly processed, and said data includes video data and/or audio data.
Still yet another aspect of the present invention is the audio-video transmitting apparatus wherein priority values can be changed in accordance with the situation by transmitting the offset value of information showing the priority for processing of said data.
A further aspect of the present invention is an audio-video receiving apparatus comprising: receiving means for receiving encoded information to which the information concerned with the priority for processing under an overload state is previously added; and priority deciding means for deciding a threshold serving as a criterion for selecting whether to process an object in said information received by said receiving means; wherein
the timing for outputting said received information is compared with the elapsed time after start of processing or the timing for decoding said received information is compared with the elapsed time after start of processing to change said threshold in accordance with the comparison result, and video data and/or audio data are or is included as said encoding object.
A still further aspect of the present invention is the audio-video receiving apparatus wherein retransmission-request-priority deciding means for deciding a threshold serving as a criterion for selecting whether to request retransmission of some of said information not received because it is lost under transmission when it is necessary to retransmit said information is included, and
said decided threshold is decided in accordance with at least one of the priority controlled by said priority deciding means, retransmission frequency, lost factor of information, insertion interval between in-frame-encoded frames, and grading of priority.
A yet further aspect of the present invention is an audio-video transmitting apparatus comprising: retransmission-priority deciding means for deciding a threshold serving as a criterion for selecting whether to request retransmission of some of said information not received because it is lost under transmission when retransmission of said unreceived information is requested is included, wherein said decided threshold is decided in accordance with at least one of the priority controlled by the priority deciding means of said audio-video receiving apparatus retransmission frequency, lost factor of information, insertion interval between in-frame-encoded frames, and grading of priority.
A still yet further aspect of the present invention is an audio-video transmitting apparatus for transmitting said encoded information by using the priority added to said encoded information and thereby thinning it when (1) an actual transfer rate exceeds the target transfer rate of information for a video or audio or (2) it is decided that writing of said encoded information into a transmitting buffer is delayed as the result of comparing the elapsed time after start of transmission with a period to be decoded or output added to said encoded information.
Another aspect of the present invention is a data processing apparatus comprising: receiving means for receiving a data series including (1) time-series data for audio or video, (2) an inter-time-series-data priority showing the priority of the processing between said time-series-data values, and (3) a plurality of in-time-series-data priorities for dividing said time-series data value to show the processing priority between divided data values; and data processing means for performing processing by using said inter-time-series-data priority and said in-time-series-data priority together when pluralities of said time-series-data values are simultaneously present.
Still another aspect of the present invention is a data processing apparatus comprising: receiving means for receiving a data series including (1) time-series data for audio or video, (2) an inter-time-series-data priority showing the priority of the processing between said time-series-data values, and (3) a plurality of in-time-series-data priorities for dividing said time-series data value to show the processing priority between divided data values; and data processing means for distributing throughput to each of said time-series-data values in accordance with said inter-time-series-data priority and moreover, adaptively deteriorating the processing quality of the divided data in said time-series data in accordance with said in-time-series-data priority so that each of said time-series-data values is kept within said distributed throughput.
Yet another aspect of the present invention is a data processing apparatus characterized by, when an in-time-series-data priority for a video is added every frame of said video and said video for each frame is divided into a plurality of packets, adding said in-time-series-data priority only to the header portion of a packet for transmitting the head portion of a frame of said video accessible as independent information.
Still yet another aspect of the present invention is the data processing apparatus wherein said in-time-series-data priority is described in the header of a packet to perform priority processing.
A further aspect of the present invention is the data processing apparatus wherein the range of a value capable of expressing said in-time-series-data priority is made variable to perform priority processing.
A still further aspect of the present invention is a data processing method comprising the steps of: inputting a data series including time-series data for audio or video and an inter-time-series-data priority showing the processing priority between said time-series data values; and
processing priorities by using said inter-time-series-data priority as the value of a relative or absolute priority.
A yet further aspect of the present invention is a data processing method comprising the steps of: classifying time-series data values for audio or video; inputting a data series including said time-series data and a plurality of in-time-series-data priorities showing the processing priority between said classified data values; and processing priorities by using said in-time-series-data priority as the value of a relative or absolute priority.
Moreover, the present invention is characterized by:
inputting, for example, a video as waveform data in accordance with the waveform-data-transmitting method; or
outputting, for example, a video as waveform data in accordance with the waveform-data-receiving.
Moreover, the present invention is characterized by:
(d) outputting the execution time of each group obtained through estimation in accordance with the waveform-data-receiving method; or
(d) inputting a data string constituted with the execution time of each group; and
(e) computing the execution frequency of each group for completing decoding within a time required to transmit a code length determined by the designation of a rate controller or the like in accordance with each execution time of the receiving means in accordance with the wave-data-transmitting method.
Furthermore, the present invention is characterized by:
(d) estimating the execution time of each group in accordance with the processing time required to encode a video and each execution frequency output by counting means; and
(e) estimating the processing time required to encode a video by using the above execution time and computing the execution frequency of each group in which the processing time does not exceed a time usable to process one sheet of picture determined by a frame rate given as the designation of a user in accordance with the waveform-data-transmitting method.
The present invention has the above structure to obtain the execution frequency of indispensable processing and that of dispensable processing, transmit the execution frequencies to the receiving side, and estimate the time required for each processing in accordance with the execution frequencies and the decoding time.
By reducing each execution frequency of dispensable processing so that the time required for decoding becomes shorter than a designated time in accordance with the estimated time of each processing, it is possible to control the decoding time to the designated time or shorter and keep a delay small.
Moreover, it is possible to set the decoding execution time to a value equal to or less than a designated time by transmitting the execution time of indispensable processing and that of dispensable processing estimated by the receiving side to the transmitting side and determining each execution frequency at the transmitting side in accordance with each execution time.
Moreover, it is possible to set the encoding estimation time to a value equal to or less than a user designated time by estimating the execution time of indispensable processing and that of dispensable processing and determining each execution frequency in accordance with each execution time and the user designated time determined by a frame rate designated by a user.